Novel Computational Techniques for Incident Reporting

Incident reporting systems help users to provide information about potential safety hazards. They, therefore, represent an important subset of the wider range of applications that support process improvement. The following pages identify a range of novel computational techniques that can be used to address problems of existing reporting systems. In particular, it is argued that computerassisted interviewing techniques, such as the familiar frame and script approaches, can guide the elicitation of incident reports. Probabilistic information retrieval systems reduce the classification problems that prevent attempts to index diverse reports in dynamic industries. Conversational casebased reasoning techniques can be used to avoid the problems of query formation that frustrate attempts to retrieve similar incidents. Finally, discourse-modeling techniques can be extended to represent the reasons why particular lessons have been learnt from particular incidents. A Brief Introduction to Incident Reporting Incident reporting applications are an instance of the more general class of systems for organizational and interorganizational learning (Weber et al, 1999). For example, the FAA’s Aviation Safety Reporting System (ASRS) was set up to provide a confidential means of learning about aviation related incidents in the United States. It, therefore, provides a primary source of information about the causes of failure. This information has been used to generate Operational Bulletins that alert the aviation industry to "timely and important" concerns in recent submissions. The ASRS also helps to disseminate best practice. The lessons learned from previous failures are disseminated through the Callback publication. Approximately 85,000 copies of this are distributed to pilots, engineers, air traffic controllers etc. However, a number of problems limit the utility of such incident reporting systems. 1. It can be difficult to elicit information about previous incidents from the users that are involved in them. Many incident reporting forms provide a cursory overview of the events leading to failure and so investigators have to visit contributors to identify missing information. This creates considerable logistical problems for the growing numbers of national and international reporting systems. 2. It can be difficult to correctly index and classify incident reports so that others can perform the statistical analyses that help to guide subsequent intervention. For example, there is a growing concern over the problems of Crew Resource Management (CRM) many aviation incident reports (Johnson, 2000a). Unfortunately, incident reports are not routinely indexed in terms of specific problems such as CRM. As a result, analysts cannot query systems such as the ASRS to retrieve every CRM related incident over the last five years (Johnson, 2000b). 3. It can be difficult to correctly issue the queries that are needed to retrieve information about particular incidents. The importance of correct query formation is illustrated by the fact that the ASRS now has a cumulative total of more than 500,000 reports. It is impossible for individuals to manually search such collections to find the lessons that apply to their systems. It is also difficult for organizations to spot emerging trends amongst the mass of data that has been collected. 4. There is a danger that rather than learning the lessons of the past, organizations will simply use incident reports to find evidence that supports their existing preconceptions and biases (Johnson, 2000b). It is, therefore, important to explain why particular lessons can be drawn from particular incidents. The following sections briefly describe a number of computational techniques that can be applied to avoid or mitigate the impact of these problems for incident reporting Systems. 2O From: AAAI Technical Report WS-00-03. Compilation copyright © 2000, AAAI (www.aaai.org). All rights reserved. Problems of Eliciting Incident Information: Computer-assisted Interviewing The problems of eliciting information about previous incidents should not be underestimated. At present many systems rely upon confidential rather than anonymous reporting. For instance, the UK CIRAS rail reporting system sends an investigator out to conduct a follow-up interview in response to every report form that is submitted. Similarly, NASA personnel go back to the contributors of many ASRS submissions. This approach requires considerable resources. There must be enough trained analysts to elicit the necessary information during follow-up visits. Alternatively, it might be possible to recruit novel computational techniques to improve the quality of information that is initially contributed in response to an incident. These techniques might, therefore, reduce the expense associated with site visits. Equally importantly, they might also avoid the biases that affect follow-up interviews. A number of social concerns must affect contributors during safety-related discussions with external interviewers. Eliciting more information in the immediate aftermath of an incident also helps to reduce any delay between the contribution of a report and a follow-up interview. The problems of extracting information from domain experts has been addressed by work on knowledge elicitation in general and by computer-aided interviewing techniques in particular (Saris, 1991). These interviewing techniques, typically, rely upon frames and scripts that are selected in response to information from the user. For example, the user of an air traffic management system might first be prompted to provide information about the stage of flight in which an incident occurred. If it happened during landing then a script associated with that stage of flight would be selected. This might provide further prompts about the activities of arrivals and departures officers or about specific items of equipment, such as MSAW protection. These detailed questions would not be appropriate for incidents during other stages of flight, such as those filed during en route operations. The relatively simple script-based techniques, described above, offer a number of further benefits. In particular, the use of computer-assisted interviewing ~:an reduce the biases that stem from the different approaches that are used by many interviewers. Inter-analyst reliability is a continuing concern in many incident report systems (Johnson, 2000b). The scripts embodied in computerassisted interviewing systems might also be tailored to elicit particular information about regulatory concerns. For instance, if previous accidents had indicated growing problems with workload distribution during certain teambased activities then scripts could be devised to specifically elicit information about these potential problems. Of course, this analysis must be balanced against the obvious limitations of computer-based interviewing techniques (Saris, 1991). Further evidence is needed to determine whether the weaknesses of computers assisted interviewing in employment selection or the analysis of consumer behavior also apply to their application in incident reporting. Until this evidence is provided then there will continue to be significant concerns about the problems of bias that can be introduced during the elicitation of information about previous failures. Problems of Indexing Dynamic Incidents: Probabilistic Information Retrieval A number of problems remain to be addressed once the information about an incident has been gathered. Perhaps the most important of these relates to the indexing of largescale collections. At present, most successful incident reporting systems rely upon relational database technology. Each incident is classified according to a number of pre-determined fields. Queries can then be constructed, using languages such as SQL, to sort, filter and combine incident data according to the information contained in these fields. This approach has a number of consequences. In particular, it can lead to an extremely static classification system because there is often no way to automatically reclassify thousands of previous incidents if changes are made to a taxonomy. For instance, many existing schemes use Reason’s (1990) GEMS taxonomy human error to classify operator behavior in the lead-up to an incident. This taxonomy has recently been revised in a number of ways. However, few of these changes have been reflected in incident reporting systems because of the costs associated with manually analyzing and reclassifying existing records. This has profound consequences. As mentioned earlier, analysts are faced with retaining distinctions that may no longer reflect the way in which particular tasks or activities are organized. Alternatively, the problems of updating previous records can result in only a small portion of the incidents having values for the most recent set of fields. In information retrieval, the concept of poor recall is used to describe situations in which only a small proportion of relevant documents are returned from a collection in response to a users’ query. Conversely, poor precision results in many irrelevant documents being returned as potential hits. These concepts have particular importance for incident reporting schemes. If the fields in a relational scheme are not updated then queries about new concepts will often result in poor precision. Users will have to construct queries from existing values that do not accurately describe the concepts or classifications that they are interested in. Conversely, adopting more dynamic classifications in which new fields will only be maintained for subsequent reports will lead to poor recall. A highly precise set of incidents can be returned, for example in